ci: guard that every bundled SKILL.md carries upstream provenance by devantler · Pull Request #41 · devantler-tech/agent-plugins

devantler · 2026-06-30T13:24:05Z

🤖 Generated by the Daily AI Assistant

Fixes #40 — Part of #38 (Theme 4: provenance & supply-chain hardening).

Problem

AGENTS.md is emphatic that bundled skills come only from upstream via gh skill install and are never hand-edited to diverge — each install records the true upstream in the skill's metadata.github-* frontmatter. But scripts/validate-manifests.sh enforced only structural parity (manifest ↔ filesystem ↔ README table). It did not assert that every bundled SKILL.md actually carries that provenance, so a hand-authored or provenance-stripped skill passed CI today — silently breaking the "sourced from upstream, never divergent" contract.

Change

validate-manifests.sh gains validate_skill_provenance(): for every plugins/*/skills/*/SKILL.md, it slices the YAML frontmatter (the block between the first two --- lines) and asserts a non-empty github-repo: line. A skill with no frontmatter, an absent key, or an empty value is rejected with a clear message. Stays jq/grep-only — no new yq dependency, matching the rest of the guard.
validate-manifests.test.sh: the fixture make_plugin now writes a SKILL.md with provenance frontmatter (happy path stays green), plus three new FAIL cases — stripped provenance, no frontmatter, and an empty github-repo value — so a future refactor that weakens the guard fails the self-test.
AGENTS.md: documents the new check (structure comment + the "Skills come from upstream" section).

Validation

shellcheck clean on both scripts.
validate-manifests.test.sh: 24 passed, 0 failed.
validate-manifests.sh against the live repo: exit 0, all currently-bundled skills carry provenance.

Behaviour-preserving for every conformant skill; only catches a genuinely hand-authored / provenance-stripped one. Opened as a draft for your promotion.

Add a provenance-presence check to validate-manifests.sh: every plugins/*/skills/*/SKILL.md must carry a non-empty metadata.github-repo in its YAML frontmatter (as gh skill install records). AGENTS.md forbids hand-authored/divergent bundled skills, but CI only enforced structural parity (manifest <-> filesystem <-> README) — a hand-edited or provenance-stripped skill passed. The guard stays jq/grep-only (no yq). Pin it in validate-manifests.test.sh: the fixture SKILL.md now carries provenance (happy path), plus three FAIL cases (stripped provenance, no frontmatter, empty github-repo). Update AGENTS.md to document the check. Fixes #40 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

coderabbitai · 2026-06-30T13:24:29Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 579f716d-5690-4441-88b9-005e68f620b4

📥 Commits

Reviewing files that changed from the base of the PR and between 3fb0101 and e7c721d.

📒 Files selected for processing (2)

scripts/validate-manifests.sh
scripts/validate-manifests.test.sh

🚧 Files skipped from review as they are similar to previous changes (1)

scripts/validate-manifests.sh

📝 Walkthrough

Walkthrough

Adds a provenance check to scripts/validate-manifests.sh for bundled SKILL.md files, updates the test fixture and negative coverage, and documents the new CI enforcement rule in AGENTS.md.

Changes

Skill provenance guard

Layer / File(s)	Summary
validate_skill_provenance() implementation and wiring `scripts/validate-manifests.sh`	Adds `validate_skill_provenance()` that parses YAML frontmatter in each bundled `plugins//skills//SKILL.md` and fails when `metadata.github-repo` is missing or empty. `main()` now runs the check after existing validations.
Fixture update and provenance test cases `scripts/validate-manifests.test.sh`	Updates the example skill fixture to include provenance frontmatter, and adds negative cases covering missing frontmatter, missing `metadata.github-repo`, empty values, and invalid placement of the key.
AGENTS.md documentation update `AGENTS.md`	Updates the repository guidance to state the new provenance guard and its CI enforcement rule.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

devantler-tech/agent-plugins#36: Adds bundled skill SKILL.md files with upstream provenance frontmatter that this new guard validates.

Suggested reviewers

botantler-1

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly states the main change: adding a CI guard for bundled SKILL.md provenance.
Description check	✅ Passed	The description matches the change set and explains the provenance guard, test updates, and documentation.
Linked Issues check	✅ Passed	The PR adds the required provenance check, expands tests, and the reported live repo run passes all bundled skills.
Out of Scope Changes check	✅ Passed	The changes are limited to the provenance guard, its tests, and related docs.
Docstring Coverage	✅ Passed	Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch claude/ci-skill-provenance-guard

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/validate-manifests.sh`:
- Around line 208-211: The provenance check in validate-manifests.sh is too
loose because it greps any github-repo line and accepts empty/commented values,
so tighten the logic to validate the metadata.github-repo field specifically.
Update the existing manifest validation block around the provenance check to
parse the actual metadata section and require a real non-empty value, rather
than matching top-level github-repo or whitespace/comment-only entries. Keep the
change localized to the skill-provenance guard in scripts/validate-manifests.sh
so it remains the source of truth.

In `@scripts/validate-manifests.test.sh`:
- Around line 200-230: The provenance regression coverage in
validate-manifests.test.sh is incomplete: it only checks missing frontmatter,
missing github-repo, and empty key values, but not misplaced or quoted-empty
provenance. Add failing fixtures around the existing bundled SKILL.md provenance
checks to assert that github-repo outside metadata is rejected and that
github-repo: "" or comment-only values are also rejected, using the same
check_fail pattern so the validate-manifests guard stays pinned.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 7843d2d5-fc80-4d5e-a095-a0d759e3ae41

📥 Commits

Reviewing files that changed from the base of the PR and between 5ad416d and 3fb0101.

📒 Files selected for processing (3)

AGENTS.md
scripts/validate-manifests.sh
scripts/validate-manifests.test.sh

…ment values Address CodeRabbit review on #41: the grep-based provenance check accepted a top-level github-repo: (outside metadata:) and treated github-repo: "" / github-repo: # comment as non-empty, so a provenance-stripped hand edit could still pass. Replace it with a single awk pass that slices the frontmatter, scopes the lookup to the metadata: block, and rejects empty, quoted-empty (""/''), and comment-only (# …) values. Stays jq/grep-free (no yq). Pin the tightening with three new self-test FAIL fixtures: top-level github-repo, quoted-empty, and comment-only. Self-test 27/0; live guard exit 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

coderabbitai Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread scripts/validate-manifests.sh Outdated

Comment thread scripts/validate-manifests.test.sh

devantler marked this pull request as ready for review June 30, 2026 13:56

devantler merged commit 9bb8a1c into main Jun 30, 2026
32 checks passed

devantler deleted the claude/ci-skill-provenance-guard branch June 30, 2026 14:13

coderabbitai Bot mentioned this pull request Jun 30, 2026

feat(gitops-kubernetes): bundle the flux-operator-mcp server (first non-skill resource) #44

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci: guard that every bundled SKILL.md carries upstream provenance#41

ci: guard that every bundled SKILL.md carries upstream provenance#41
devantler merged 2 commits into
mainfrom
claude/ci-skill-provenance-guard

devantler commented Jun 30, 2026

Uh oh!

coderabbitai Bot commented Jun 30, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

devantler commented Jun 30, 2026

Problem

Change

Validation

Uh oh!

coderabbitai Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai Bot commented Jun 30, 2026 •

edited

Loading